Search for: All records

Medical cyber-physical systems (MCPS) are increasingly adopting learning-enabled components (LECs) to enhance their decision-making capabilities. Due to the safety-critical nature of MCPS, these systems must maintain high performance on both expected and unexpected input data. Therefore, ensuring the robustness of LE-MCPS is crucial for their successful deployment. Existing research predominantly focuses on robustness tosynthetic adversarial examples, crafted by adding imperceptible perturbations to clean input data. However, these synthetic adversarial examples do not accurately reflect the most challenging real-world scenarios, especially in the context of healthcare data. Consequently, robustness to synthetic adversarial examples may not necessarily translate to robustness againstnaturally occurring adversarial examples. We propose a method to evaluate the robustness of LE-MCPS to natural adversarial examples. The method curates naturally adversarial datasets leveraging probabilistic labels obtained from automated weakly supervised labeling which combines noisy and cheap-to-obtain labeling heuristics. Based on these labels, the method adversarially orders the input data and uses this ordering to construct a sequence of increasingly adversarial datasets for assessing robustness. Our evaluation on six MCPS case studies and two non-medical case studies demonstrates (1) the efficacy and statistical validity of our approach to generating naturally adversarial datasets and (2) the utility of our robustness evaluation in classifying robust and non-robust LE-MCPS. 

Classification of clinical alarms is at the heart of prioritization, suppression, integration, postponement, and other methods of mitigating alarm fatigue. Since these methods directly affect clinical care, alarm classifiers, such as intelligent suppression systems, need to be evaluated in terms of their sensitivity and specificity, which is typically calculated on a labeled dataset of alarms. Unfortunately, the collection and particularly labeling of such datasets requires substantial effort and time, thus deterring hospitals from investigating mitigations of alarm fatigue. This article develops a lightweight method for evaluating alarm classifiers without perfect alarm labels. The method relies on probabilistic labels obtained from data programming—a labeling paradigm based on combining noisy and cheap-to-obtain labeling heuristics. Based on these labels, the method produces confidence bounds for the sensitivity/specificity values from a hypothetical evaluation with manual labeling. Our experiments on five alarm datasets collected at Children’s Hospital of Philadelphia show that the proposed method provides accurate bounds on the classifier’s sensitivity/specificity, appropriately reflecting the uncertainty from noisy labeling and limited sample sizes.